Abstract Title Page 

Not included in page count. 



Title: Teacher Training, Classroom Consultation, and Child Outcomes in the Foundations of 
Learning Project 

Author(s): Chrishana M. Lloyd, Ph.D., MDRC, Megan Millenky, M.P.P., MDRC 



2011 SREE Conference Abstract Template 




Abstract Body 



Background / Context: Description of prior research and its intellectual context. 

Long-term studies indieate that quality preschool services offer significant benefits to 
disadvantaged children in the areas of lifetime educational attainment, earnings, and employment 
(Schweinhart et ah, 2005; U.S. Dept, of Health and Human Services, 2003). Research also 
suggests that many children from disadvantaged backgrounds lack the social and emotional skills 
necessary to thrive in the preschool environment (Blanchard, Gurka, & Blackman, 2006; 
Fantuzzo, Bulotsky, McDermott, Mosca, & Lutz, 2003; Qi & Kaiser, 2003; Raver & Knitzer, 
2002). Children with low levels of social competence and high rates of behavior problems 
represent an especially large percentage of school districts’ high-expenditure pupils (Chambers, 
Kidron, & Spain, 2004). These children are more likely to repeat a grade early in elementary 
school (Beebe-Frankenberger, Bocian, MacMillan, & Gresham, 2004), and to receive special 
education services (National Center for Health Statistics, 2005; Wagner & Blackorby, 2002). 
They are also expelled at higher rates (Gilliam, 2005), and in addition to limiting their own 
learning, children with behavioral challenges inhibit the learning of peers by distracting and 
diverting teacher attention away from other students (Arnold, McWilliams, & Arnold, 1998; 
Raver, 2002). 

Evidence from teachers supports this concern; preschool teachers in low-income neighborhoods 
report that between 15 and 30 percent of the young children in their classrooms exhibit clinically 
high levels of disruptive and challenging behaviors (Gross, Sambrook, & Fogg, 1999; 
Kuperschmidt, Bryant, & Willoughby, 2000; Rimm-Kaufman, Pianta, & Cox, 2000). Moreover, 
a nominal number (between 1-10%) receive services for these difficulties (Kazdin & Kendall, 
1998; Pottick & Warner, 2002). 

In general, educators have limited knowledge and receive very little training about how to 
address these issues (Weist, 2005). This is unfortunate since early childhood communities and 
schools are a logical choice for the delivery of supports to address children’s challenging 
behavior. Emerging research in the prevention literature suggests that improving children’s 
social and emotional skills may reduce problem behaviors (Raver, Jones, Ei-Grining, Zhai, 
Metzger & Dub, 2009) and be an effective way to improve school readiness (Morris, Raver, 
Eloyd,& Millenky, 2009). In fact, teachers spend an average of 1080 hours per year with 
children (Weist, 2005), and this time lends itself to teachers engaging with children in a variety 
of ways including supporting their social and emotional development (Han Weiss, 2005; Durlack 
and Wells, 1997). 

Purpose / Objective / Research Question / Focus of Study: Description of the focus of the research. 

This presentation will answer the following research questions: 

What was the intended implementation of the training and coaching component of EOE? 

To what extent was the training and consultation implemented as intended? 

How did the combined training and coaching model utilized in the Eoundations of Teaming 
(EOE) project influence children’s behavior and approaches to learning? 
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Setting: Description of the research location. 



FOL emerged from early lessons from CSRP, a randomized trial of 18 Head Start centers 
directed by Dr. Cybele Raver that tested the effectiveness of a uniquely designed two-pronged 
integrated strategy that combined teacher training and classroom consultation (most models only 
utilize one of these methods). The FOL project was launched in Newark, NJ, in two phases: a 
smaller-scale pilot phase during the 2006-2007 school year, followed by a second larger scale 
phase in the 2007-2008 school year. In Chicago, IL, the project was launched during the 2008- 
2009 academic year. 

Population / Participants / Subjects: Description of the participants in the study: who, how many, key 
features or characteristics. 

The full scale FOL project in Newark included a total of 51 Head Start, public school, and 
privately operated preschool programs. Principals and Site Directors were asked to choose a 
single four-year-old classroom within their school or center to participate and each of the 5 1 
classrooms was randomly assigned to either the treatment or control condition In Chicago, 
directors in 20 Head Start preschool programs selected two classrooms to participate and then 
random assignment was conducted at the program, or center, level. The Chicago classrooms 
included a greater mix of three- and four- year old children than in Newark. Approximately 600 
children participated in each site. 



Intervention / Program / Practice: Description of the intervention, program or practice, including details 
of administration and duration. 

Throughout the course of an academic year (September 2008-May 2009 in Chicago), treatment 
classrooms received the following: 

1 . Teacher Training: Lead and Assistant Teachers were invited to attend five six-hour training 
sessions designed to provide teachers with concrete strategies for addressing challenging 
behaviors and developing positive relationships with their students. The training was based 
on Carolyn Webster Stratton’s Incredible Years curriculum and included role-playing, 
vignettes, small-group discussions, and structured planning time for teachers. 

2. Classroom Consultation: Treatment classrooms were assigned a Master’s level Clinical 
Classroom Consultant (CCC) to work with them on managing challenging behavior in the 
classroom. The goal of the classroom consultation model was to provide a resource to help 
teachers implement the strategies learned at the training and to provide one-on-one clinical 
interventions for children at highest risk. 

3. Stress Management: During the winter, the CCC and the CCC Coordinator provided 
customized Stress Management Workshops for all program teachers. The workshops were 
designed to help teachers identify sources of stress and develop concrete strategies for 
addressing them while maintaining a positive atmosphere in the classroom. In the months 
leading up to and following the workshop, the CCCs helped reinforce the stress management 
skills and techniques with teachers. 
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Research Design: Description of research design (e.g., qualitative case study, quasi-experimental design, 
secondary analysis, analytic essay, randomized field trial). 



The FOL project used a randomized experimental-control design to assess the impact of the 
model, the program group received the full, integrated intervention combining teacher training 
with classroom consultation, and the control condition continued their standard program 
operations without any enhanced services. In Newark, randomization occurred within blocks; in 
Chicago, randomization was conducted in matched pairs. In addition, a qualitative research 
component was included to better understand the implementation and replication processes for 
the intervention. The qualitative component included attention to the dosage and quality all of 
the intervention components including training, coaching, and stress management services. 

Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 

In the Chicago findings that are the center of the analysis presented here, the implementation 
analysis triangulated data from qualitative interviews and focus groups with teachers and 
consultants, teacher surveys and evaluations, classroom observations, field notes, and research 
meetings along with data from a structured template completed by consultants on a weekly basis 
(the Service Provision Form) to draw conclusions about implementation dosage and quality for 
all of the intervention components. Dosage data (including Service Provision Form data) was 
analyzed using SAS. Interview transcripts were transcribed and subsequently coded in NVivo 
using a constant comparative method (Glaser and Strauss, 1967) which allowed key themes to 
emerge. To assure reliability in coding two members of the team met regularly to develop and 
review the initial codes, generate data categories, and review interpretations of the data. 

The impact analysis draws from baseline and follow-up teacher reports of children’s behavior 
and self regulation in the classroom and direct child assessments conducted in the spring of the 
intervention year. In the analyses presented, regression-adjusted means are compared for FOL 
and control classrooms and children. Controls for matched-pair assignment are included in all 
regression models. In addition, a grand mean imputation strategy was used for missing site, 
teacher and student baseline characteristics in this analysis. 

Random assignment at the center level and inclusion of two classrooms per participating Head 
Start center necessitated that a three level model would be used for tests of program impacts on 
child level outcomes. The child-level regression includes controls for matched-pair assignment, 
baseline child and teacher characteristics, and a limited number of baseline site characteristics. 
The model for the teacher outcomes adds additional teacher covariates to control for teacher 
characteristics that could have affected teachers’ reporting on these outcomes 
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Findings / Results: Description of the main findings with specific details. 

Preliminary analysis of dosage and implementation data indieates the following: 

• We find high levels of dosage of the teacher training. The large majority of lead and 
assistant teaehers attended the full dosage of the program. 

• Ratings of teacher satisfaction with the trainings were quite high. Exit surveys 
conducted at the end of each training session found that the average rating ranged from 
4.2 to 4.6 (out of a 1 to 5 rating system) that the teachers strongly agreed that the training 
content was clear, the training environment was conducive to learning, the trainers 
themselves were effective and clear, and the training enhanced the teachers’ professional 
development. 

• There were high levels of classroom consultation. Clinical Classroom Consultants 
(CCCs) were placed for one day per week in treatment classrooms. Consultants provided 
an average 217 hours of consultation services per classroom over the course of the 
academic year out of an expected 253 hours. 

• Consultants were critical in supporting intervention implementation. The 
collaboration between consultants and teachers (in addition to an evidenced based 
training approach and stress management) appears to offer significant advantages to 
supporting implementation. 

• Manualization and technical assistance around the consultation model was key. 

CCCs benefitted from written guidance about the stages of intervention implementation, 
and found value in being able to not only access clinical supervision but also support 
around intervention fidelity issues in order to conduct their jobs successfully. 

Preliminary analyses of outcomes for children in Chicago indicate the following: 

Impacts of FOL-Chicago varied across outcomes assessed, as well as hy the source of that 
information (See Tables 1, 2 and 3). More specifically, we find some evidence of positive 
impacts of the model on observer reports and direct assessments of children’s approach to 
learning (positive engagement, executive function, and self-regulation- see Table 1), but less 
consistent effects for outcomes less centrally targeted by the intervention (social problem solving 
skills and academic skills- see Table 2). And, information collected from teachers shows no 
benefits of the program on both targeted and nontargeted outcomes; even though information 
collected from assessors does show some benefits of the intervention (see Table 3). 

Impacts from FOL-Chicago are summarized in Figure 1 in the context of findings from CSRP 
and FOL-Newark, as they build off of the findings from the earlier two trials. Favorable effects 
are denoted by + signs, unfavorable by - signs, no effect as 0, and if the construct was not 
measured, as n.a. or not available. 

Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 

The implementation and impact findings in Chicago, coupled with the earlier analysis of the 
Chicago School Readiness Project and FOL-Newark, indicate a positive benefit to classrooms 
and students for the combined training and consultation model. Classrooms appear to be better 
managed by teachers, and students are more positively engaged and exhibit less conflictual 
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behavior. Notably, having an explicit and manualized process for consultation and clear 
standards and access to support about the various phases of consultation, was a critical driver of 
the intervention implementation process and provided a standard from which to understand if 
consultation occurred as intended. 

However, the data varied slightly across these studies and follow up data is limited in FOL. 
Further studies would benefit from a consistency in measures across sites and include pre and 
post measurement of both classrooms and children. As mentioned in the finding sections, 
independent observations and assessments appear to be less biased, more fine-tuned instruments 
for illustrating the differences across groups of children and classrooms. In addition, while there 
has been some focus and follow up on outcomes for children beyond the intervention year, 
further data collection on the impact of the model on teachers in subsequent years could more 
fully illustrate the model’s potential impact. 
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Appendix B, Tables and Figures 



Foundations of Learning Demonstration 
Figure 1 

Summary of Findings across Studies: CSRP, FOL-Newark (FOL-N), and FOL-Chicago (FOL-C) 



Directly Targeted Outcomes Intermediate Classroom Outcomes First-order Child Outcomes 



Second-order Child Outcomes 



Positive Classroom 
Management 

CSRP + 

FOL-N + 

FOL-C n.a 



Classroom 

Productivity 

CSRP 

FOL-N 

FOL-C 



n.a. 

+ 



Quality of Instruction 




CSRP 


n.a. 


FOL-N 


0 


FOL-C 


n.a. 



Student Problem Behavior 

Conflict/Externalizing Behavior Problems 
(Observed) 



CSRP + 

FOL-N + 

FOL-C n.a. 

Conflict/Externalizing Behavior Problems 
(Teacher Report) 

CSRP + 

FOL-N 0 
FOL-C 0 



student Approaches to Learning 


Positive Engagement 


CSRP 


0 




FOL-N 


+ 




FOL-C 


+ 


Attentiveness/Behavior Control 


CSRP 


+ 




FOL-N 


+ 




FOL-C 


0 


Executive Functioning 


CSRP 


+ 




FOL-N 


n.a. 




FOL-C 


+ 


Effortful Control 


CSRP 


0 




FOL-N 


n.a. 




FOL-C 


+ 



Student Positive 




Social Behavior 




CSRP 


n.a. 


observed FOL-N 


0 


FOL-C 


n.a. 



Student Early Academic 
Skills 

CSRP + 
FOL-N n.a. 
FOL-C + 



student Social Problem- 


Solving Strategies 




CSRP 


n.a. 


FOL-N 


n.a. 


FOL-C 


0 
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Foundations of Learning 
Table 



Program Impacts on Directly Targeted Outcomes: Approaches to 
Child Behavior Observed and 



Outcom 


FOE 

Only 

Differenc 

(Impact 


Standar 

Erro 


Effec 
Siz P 




Aonroaches to 










Positive engagement/emotion 


0.14** 


0.03 


0.34 


0.01 


Attentiveness/inhibitory control 


0.04 


0.09 


0.07 


0.66 


Executive Functioning (Fine Motor, 


0.03 


0.05 


0.07 


0.65 


Executive Functioning (Gross Motor, 


2.38* 


1.06 


0.38 


0.06 


Effortful Control 


7.92* 


3.76 


0.32 


0.07 


Sample 


308 









SOURCES: MDRC calculations using Interviewer Assessment observations and direct child assessments 
Apri -May 



NOTE : Statistical levels are indicated as: *** = 1 percent; ** = 5 percent; * = 10 

The table estimates of program impact in a model that adjusts for random assignment matched 

child's teacher reported baseline Behavior Problem Index score, child gender and age, teacher age and level 
education, additional site level staff present, and site level percentage of children with single 

"Positive engagement/emotion" and "Attentiveness/inhibitory control" refer to observer reports of 
dimension on the PSRA Assessor Report. "Executive Functioning, Fine Motor" refers to percent correct on 
"Pencil Tap" measure, "Executive Functioning Gross Motor refers to scores on the "Head to Toes" 



2011 SREE Conference Abstract Template 



B-2 




Foundations of Learning Demonstration 
Table 2 

Program Impacts on Fre- Academic Skills and Social Problem- solving Skills 



Outcome 


FOL Chicago 
Only 

Difference 

(Impact) 


Standard 

Error P-value 


Pre-Academic skills 


Early literacy (assessec^ 


2.36 


1.95 0.27 


Early math (assesse($ 


5.10 ** 


1.81 0.03 


Social Problem-solviii£ skills 


Adaptive problem solving (assessec^ 


0.11 


0.37 0.77 


Sample size 


308 





SOURCES: MDRC calculations using direct child assessm entsinAprilrMay 2009 



ROTES: Statistical sigrificance levels are indicated as: ***= 1 percent; **= 5 percent; *= 10 
percent. 

The table presents estimates of pro gram imp act in a mo del that adjusts for random assignment 
matchedpairs, child's teacher reported baseline B ehavior Problem Index score, child gender and 
age, teacher age and level of education, additional site level staff present, and site level 
percentage of children with sin^e parents 

"Early literacy" refers to score on the Letter-Word I dentificationsubte stand "Early math" 
refers to score on the AppEed Prolems subtest of the Woodcock Johnson. "Early language" refers 
to score on Peabody Picture Vocabulary Test. "Adaptive problem solving composite" refers to 
the Challen^g Situations Task (C ST). 
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Foundations of Learning Demonstration 
Table 3 

Program Impacts on Teacher Reported Child Outcomes 



Outcome 


FOL Chicago 
Only 

Difference 

(Impact) 


Standard 

Error 


Effect 

Size 


P-value 


Problem behavior 


Internalizing problems 


D.99 ** 


0.42 


0.41 


0.05 


Externalizing problems 


Q.76 


0.92 


0.13 


0.44 


Teacher-Student conflict 


-0.41 


1.28 


-0.07 


0.76 


Attention problems 


0.61 


1.21 


0.13 


0.63 


AoDroaches to leamins 

Work-Related skills 


-0.12 


0.19 


-0.10 


0.56 


Positive social behavior 


Complrance with teachers' directives 


-0.07 


0.15 


-0.09 


0.68 


Social competence 


-0.13 


0.14 


-0.22 


0.40 


Teacher-Student closeness 


-0.08 


1.30 


-0.02 


0.95 


Pre- Academic skills 


Early academic skills 


-0.19 


0.52 


-0.16 


0.29 


Sample size 


526 









SOURCE: B a se d on MDRC calculationsfrom responses to teacher survey in September 2D0S and 
ApritMay 2009. 



NOTES: Statistical significance levels are indicated as: ***= 1 percent, **= 5 percent; *= 10 percent. 

The table presents estim ates of prog:am impact in a m o del that adjust for random assi^rm ent 
matchedpairs, child's teacher reported baseline Behavior Problems Index (BPI) score, child gender and 
age, teacher age, teacher level of e due ation,te a cherpsychologcal distress, teacherbominthe U.S., 
teacher confidence in job skills, a dditional site level staff present, and site level percentage of children 
with sin^e p arents. 

Outcome c ontrols f or the child's baseline score on a given measure, when available. These include 
baseline measures forCooperFarranBehavioralRatingsScale, BPI, andPositive Behavior Sc ale. 

"Internalizing problems" and "Externalizing problems" refer to the internalizing and externalizing 
sub sc ales of the BPI . BPI internalizing and externalizing scaleswere create dbased on factor analysis 
work. "Teacher-Student conflict" and "Teacher-Student closeness" refer to the Student-Teacher 
Relationshrp Scale conflict and closeness subscales. "Attention problem s" refers to the C aregver- 
TeacherReportForm attentionproblem s subscale. "Compliance with teacher^ directives" and "Social 
competence" referto the Positive B ehaviorScale compliance and social competence subscales. "Work- 
Related dolls" refers to the C ooper FarranB ehavioralRatings Scale work-related dcills subscale. 

"Early academic skills" refers to the Academic RatingSeale reports of teacher^ perceptions of 
children's language, literacy, math and general knowledge skills. 
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